Steerable High-Resolution Display

ABSTRACT

A display system comprising a foveal display having a monocular field of view of at least 1 degree is positioned within a scannable field of view of at least 20 degrees, the foveal display positioned for a user. In one embodiment, the foveal display is positioned for the user&#39;s fovea.

RELATED APPLICATION

The present application claims priority to U.S. Provisional ApplicationNo. 62/477,404, filed on Mar. 27, 2017, and U.S. Provisional ApplicationNo. 62/575,354, filed on Oct. 20, 2017, and incorporates bothapplications in their entirety.

BACKGROUND

Near-eye displays have the competing requirements of displaying imagesat a high resolution, over a large field of view (FOV). For manyapplications in virtual and augmented reality, the field of view shouldbe greater than 90 degrees, and ideally the binocular field of viewwould extend past 180 degrees. At the same time, the resolution of thedisplay should match that of the human visual system so that little orno pixelation is perceived in the virtual images. Combining these tworequirements in a single system presents a number of challenges. Toavoid the appearance of pixelation, the resolution needs to be on theorder of 0.01-0.02 degrees per pixel. Over a 90-degree square field ofview, this corresponds to 4.5 k×4.5 k pixels per eye or higher.Achieving such resolutions is challenging at the level of the panel, thedrive electronics, and the rendering pipeline.

Additionally, optical systems that can project wide FOV images to theuser with sufficiently high resolution over the entire field of view arealso difficult to design. Systems architectures that are able to presentthe user with high resolution images over a wide field of view, whilesimultaneously reducing the rendering, data rate, and panel requirementswill enable new applications for augmented and virtual reality systems.

LIST OF FIGURES

The present invention is illustrated by way of example, and not by wayof limitation, in the figures of the accompanying drawings and in whichlike reference numerals refer to similar elements and in which:

FIG. 1A illustrates an eye showing the fovea.

FIG. 1B illustrates the vision ranges of the eye.

FIG. 1C illustrates the relationship between distance from the foveacenter and visual acuity.

FIG. 1D illustrates an exemplary a vertical field of view.

FIG. 1E illustrates an exemplary horizontal field of view.

FIG. 1F illustrates an eye with a first exemplary gaze vector.

FIG. 1G illustrates the position of the steerable foveal display for thefirst exemplary gaze vector shown in FIG. 1F.

FIG. 1H illustrates an eye with a second exemplary gaze vector.

FIG. 1I illustrates the position of the steerable foveal display for thesecond exemplary gaze vector shown in FIG. 1H.

FIG. 1J illustrates one embodiment of the binocular display showingfoveal displays and field displays for each eye.

FIG. 1K illustrates one embodiment of the binocular display showingfoveal displays for each eye and a shared field display.

FIG. 2 is a block diagram of one embodiment of the system.

FIG. 3 is an illustration of one embodiment of the movement of the highresolution area, over time, in a moveable foveal display.

FIGS. 4A and 4B are a flowchart of one embodiment of utilizing thedisplay.

FIG. 5A is an illustration of one embodiment of a hybrid display.

FIG. 5B is an illustration of one embodiment of a display using roll-offmagnification.

FIG. 6 is an illustration of one embodiment of a hybrid display.

FIG. 7 is an illustration of one embodiment of a hybrid display.

FIG. 8 is an illustration of one embodiment of a hybrid display.

FIG. 9 is an illustration of one embodiment of a hybrid display.

FIG. 10 is an illustration of one embodiment of a hybrid display.

FIG. 11 is an illustration of one embodiment of a hybrid display.

FIG. 12 is an illustration of one embodiment of a hybrid display usingtime multiplexing.

FIGS. 13A and 13B are an illustration of one embodiment of a hybriddisplay using time multiplexing.

FIGS. 14A and 14B are an illustration of one embodiment of a fovealdisplay using a waveguide which may be used in the systems above.

FIGS. 15A and 15B is an illustration of one embodiment of a fielddisplay using a waveguide which may be used in the systems above.

FIG. 16A is an illustration of another embodiment of a hybrid displaysystem.

FIG. 16B is an illustration of another embodiment of a hybrid displaysystem.

FIG. 17 is a flowchart of one embodiment of using the foveal displaywith an external display.

FIG. 18 is a flowchart of one embodiment of using a foveal display withno associated larger display.

FIG. 19 is a flowchart of one embodiment of blending edges of the fovealdisplay.

FIG. 20 is a flowchart of one embodiment of using eye movementclassification.

FIG. 21 is a table of exemplary types of eye movements.

FIG. 22 is a flowchart of one embodiment of smart positioning.

FIG. 23 is a block diagram of one embodiment of a computer system thatmay be used with the present invention.

DETAILED DESCRIPTION

The present application discloses a steerable foveal display, referredto as a foveal display. The foveal display in one embodiment ispositioned to provide the high resolution area where the user's fovea islocated. The “fovea” is small depression in the retina of the eye wherevisual acuity is highest. FIG. 1A illustrates the eye, showing theretina and the fovea. The center of the field of vision is focused inthis region, where retinal cones are particularly concentrated. Thecenter of the fovea is the region of the retina with the highestresolution but has a field of view which is around 2 degrees. Theregions of visual acuity, ranging from the highest resolution fovealregion to the lowest resolution far peripheral region, are illustratedin FIG. 1B. The resolution of the eye decreases by almost an order ofmagnitude farther than 20 degrees away from the center of the fovea.FIG. 1C illustrates the drop-off in acuity (Snellen fraction) based onthe distance from the center of the fovea (eccentricity).

In one embodiment, the system takes advantage of this by providing asteerable foveal display directed to align with the center of the fieldof view of the user's eye, or another calculated position. In oneembodiment, a field display provides a lower resolution field displayimage over a larger field of view. This means that the user perceivesthe images in their peripheral vision, as well as in the direction oftheir gaze. In one embodiment, the system provides a high resolutionimage using a foveal display, directed primarily toward the center ofthe field of view of the user's eye, and a field display image over alarge field of view utilizing a second field display. This means thatthe user perceives the images in their peripheral vision, as well as inthe direction of their gaze. In one embodiment, the system uses a highpixel density display per eye to present a high resolution image over asmall field of view and a lower-resolution image over a large field tofill in the binocular and peripheral regions. In one embodiment, theresolution of the foveal display is between 0.2 arc-minutes per pixeland 3 arc-minutes per pixel. In one embodiment, the resolution of thefield display is between 1 arc-minutes per pixel and 20 arc-minutes perpixel. In one embodiment, the field display and foveal display may becombined in a single variable pixel display. In one embodiment, thesystem uses a variable pixel density display for each eye to present ahigh resolution image over a small field of view to the foveal regionsof each eye and a lower resolution image over a large field to fill inthe binocular and peripheral regions. In one embodiment, the variablepixel density display may be a standard display addressed at a variabledensity.

Such a system creates the perception of a high resolution image with awide field of view while requiring only a fraction of the number ofpixels or amount of processing of a traditional near-eye display ofequally high perceived resolution. In one embodiment, such a system alsoreduces the power consumption of the rendering system significantly byreducing the number of pixels rendered.

The system may include more than two displays, in one embodiment. In oneembodiment, there may be three levels of resolution, covering the fovealarea for each eye, the area of binocular overlap, and the peripheralarea. In one embodiment, the video images for multiple displays andresolutions may be aggregated together. In another embodiment, the videoimages for multiple displays and resolutions may be separate.

FIG. 1D illustrates an exemplary vertical field of view, showing the55-degree area of focus, or comfort zone, as well as the peripheralareas. The symbol recognition area is approximately 30 degreesvertically.

FIG. 1E illustrates an exemplary horizontal field of view, showing a60-degree area of focus, and 30-degree symbol recognition zone, as wellas the peripheral vision areas, and the full binocular range of 135degrees. Beyond that, there is a monocular range (for the right and lefteyes), and a temporal range which is only visible when the user shiftsthe eye.

In one embodiment, the steerable foveal display is positioned within thevertical and horizontal 30-degree symbol recognition area. In anotherembodiment, the steerable foveal display is positioned within the55-degree vertical and 60-degree horizontal area of focus/comfort zone.

FIGS. 1F and 1G illustrate the fields of view of the foveal display forone eye. In one embodiment, the foveal display 110 is positioned to becentered around the gaze vector 105. The gaze vector defines the centerof the eye's field of view.

In one embodiment, the field of view of the foveal display 110 is amonocular field of view of a minimum field of view spanning 1 degree anda maximum field of view spanning 20 degrees. The field of view of fielddisplay 120 in one embodiment provides a monocular field of viewspanning 40 degrees, and at most the full monocular range. The fullmonocular range of the field of view is typically considered to be 60degrees toward the nose, 107 degrees away from the nose, and 70 degreesabove the horizontal, and 80 below the horizontal.

In one embodiment, a field display 120 may provide image data outsidethe range of the foveal display 110. FIG. 1F provides a top view,showing the eye, and the field of view of a foveal display 110 centeredaround the gaze vector 105. FIG. 1G provides a front view, showing theexemplary position of the field of view of the foveal display 110. Inone embodiment, a foveal display 110 has a total scannable field of view160 between 20 and 160 degrees, within which it can be positioned. Asnoted above, the foveal display 110 has a monocular field of view of atleast 1 degree. In one embodiment, the foveal display foveal 110 has amonocular field of view of 10 degrees, and the total scannable field ofview 160 for the foveal display is 60 degrees. This enables thepositioning of the foveal display 110 at the correct location.

FIGS. 1H and 1I show the field of view of the foveal display 110positioned in a different location, as the user is looking up and to theleft. As can be seen in this configuration, the field of view of thefoveal display 110 is moved, and the portion of the field of view of thefield display above and below the field of view of the foveal display110 is not even. FIG. 1I shows an exemplary positioning of the field ofview of the display from the front.

Using a system including a foveal display in combination with a fielddisplay creates the perception of a high resolution image with a widefield of view while requiring only a fraction of the number of pixelsand amount of processing of a traditional near-eye display. In oneembodiment, such a system also reduces the power consumption of therendering system significantly by reducing the number of pixelsrendered.

The system may include more than two displays per eye, in oneembodiment. In one embodiment, there may be three levels of resolution,covering the foveal area for each eye, the area of binocular overlap,and the peripheral area. In another embodiment, the system includes onlythe steerable foveal display, and the field display may be provided byan external system. In another embodiment, the system may consist onlyof the steerable foveal display with no associated field display.

FIG. 1J illustrates one embodiment of the binocular display includingthe field of view of the right eye foveal display 110A, and the field ofview for the left eye foveal display 110B. For each of the right eye150A and left eye 150B, there is also a field display with a largerfield of view, 120A and 120B respectively. The field display field ofview 120A, 120B, in one embodiment extends through at least the area offocus.

FIG. 1K illustrates one embodiment of the binocular display includingthe field of view of the right eye foveal display 110A and the field ofview for the left eye foveal display 110B. In this configuration,however, the field display 170 is a single display that extends acrossthe user's field of view. In one embodiment, the foveal display and thefield display may be a display integrated into wearable display, such asgoggles. In another embodiment, the foveal display may be part of awearable device, while the field display is a separate display such as aprojector or screen.

FIG. 2 illustrates one embodiment of the exemplary optical system 210,280 and associated processing system 238. In one embodiment, theprocessing system may be implemented in a computer system including aprocessor. In one embodiment, the processing system 238 may be part ofthe display system. In another embodiment, the processing system 238 maybe remote. In one embodiment, the optical system 210, 280 may beimplemented in a wearable system, such as a head mounted display. Thefoveal image is presented to the user's eye through a right eye fovealdisplay 220 and left eye foveal display 230, which direct the fovealdisplay. In one embodiment, the foveal displays 220, 230 direct thefoveal display image primarily toward the center of the field of view ofthe user's eye. In another embodiment, the image may be directed to adifferent location, as will be described below.

The foveal image for the right eye is created using a first displayelement 222. In one embodiment, the display element is a digitalmicromirror device (DMD). In one embodiment, the display element 222 isa scanning micromirror device. In one embodiment, the display element222 is a scanning fiber device. In one embodiment, the display elementis an organic light-emitting diode (OLED). In one embodiment, thedisplay element 222 is a liquid crystal on silicon (LCOS) panel. In oneembodiment, the display element 222 is a liquid crystal display (LCD)panel. In one embodiment, the display element 222 is a micro-LED ormicro light emitting diode (μLED) panel. In one embodiment, the displayelement is a scanned laser system. In one embodiment, the system is ahybrid system with an off axis holographic optical element (HOE). In oneembodiment, the system includes a waveguide. In one embodiment, thewaveguide is a multilayer waveguide. In one embodiment, the displayelement may include a combination of such elements. FIGS. 5-16 belowdiscuss the display elements in more detail.

In one embodiment, the first display element 222 is located in anear-eye device such as glasses or goggles.

The focus and field of view for the foveal display is set usingintermediate optical elements 224. The intermediate optical elements 224may include but are not limited to, lenses, mirrors, and diffractiveoptical elements. In one embodiment, the focus of the virtual image isset to infinity. In another embodiment, the focus of the virtual imageis set closer than infinity. In one embodiment, the focus of the virtualimage can be changed. In one embodiment, the virtual image can have twoor more focal distances perceived simultaneously.

In one embodiment, the foveal display image is directed primarily towardthe center of the field of view of the user's eye. In one embodiment,the field of view (FOV) of the foveal display image is greater than 1degree. In one embodiment, the FOV of the foveal display image isbetween 1 degree and 20 degrees. In one embodiment, the foveal displayimage may be larger than 5 degrees to address inaccuracies in eyetracking, provide the region needed to successfully blend such that theuser cannot perceive the blending, and account for the time it takes toreposition the foveal display for the various types of eye movements.

In one embodiment, the system further includes a lower resolution fielddisplay image, which has a field of view of 20-220 degrees.

In one embodiment, the foveal display image is projected directly ontothe user's eye using a set of one or more totally or partiallytransparent positioning elements 226. In one embodiment, the positioningelements 226 include a steerable mirror. In one embodiment, thepositioning elements 226 include a curved mirror. In one embodiment, thepositioning elements 226 include a Fresnel reflector. In one embodiment,the positioning elements 226 include a diffractive element. In oneembodiment, the diffractive element is a surface relief grating. In oneembodiment, the diffractive element is a volume hologram. In oneembodiment, the display 220 may include a focal adjustor 223, whichenables the display to show image elements at a plurality of focaldistances in the same frame. In one embodiment, the focal adjustor 223may be an optical path length extender, as described in U.S. patentapplication Ser. No. 15/236,101 filed on Aug. 12, 2016.

A similar set of elements are present for the left eye foveal display230. In one embodiment, the right eye foveal display 220 and the lefteye foveal display 230 are matched. In another embodiment, they mayinclude different elements.

In one embodiment, an eye tracker 240 tracks the gaze vector of theuser, e.g. where the eye is looking. In one embodiment, the eye trackingsystem is a camera-based eye tracking system 240. In one embodiment, eyetracking system 240 is an infrared scanning laser with a receivingsensor. Other eye tracking mechanisms may be used. Foveal positioncalculator 245 determines a center of the user's field of view based ondata from the eye tracking system 240.

In one embodiment, the adjustable positioning elements 226, 236 are usedto adjust the foveal display 220, 230 to position the foveal image to bedirected primarily toward the center of the field of view of the user'seye. In one embodiment, the direction of the image is adjusted bychanging the angle of a mirror, one of the position elements 226, 236.In one embodiment, the angle of the mirror is changed by usingelectromagnetic forces. In one embodiment, the angle of the mirror ischanged by using electrostatic forces. In one embodiment, the angle ofthe mirror is changed by using piezoelectric forces. In one embodiment,the adjustable element is the image source, or display element 222, 232which is moved to position the image. In one embodiment, the fovealimage is positioned to be directed to the center of the field of view ofthe user's eye. In another embodiment, another position element 226, 236may be changed, such as a steering element 226, 236.

A field display 280 communicates with the processing system 238 viacommunication logics 270, 290. In one embodiment, there may be multipledisplays. Here, two field displays are indicated, field display 285 andperipheral display 288. Additional levels of resolution may also beshown. In one embodiment, the field display 280 may include a singlefield display 285 viewed by both eyes of the user, or one field displayper eye. In one embodiment, the field display 280 may have variableresolution.

In one embodiment, when the field display 280 is a separate system, syncsignal generator 292 is used to synchronize the display of theindependent foveal display 210 with the display of the field display280. In one embodiment, the sync signal generator 292 is used tosynchronize the adjustable mirror, or other positioning element of thefoveal display with the field display. This results in thesynchronization of the displays. In one embodiment, field display 280includes blender system 294 to blend the edges of the foveal displayimage with the field display image to ensure that the transition issmooth.

In one embodiment, the lower resolution field display image is presentedto the user with a fully or partially transparent optical system. In oneembodiment, this partially transparent system includes a waveguideoptical system. In one embodiment, this partially transparent systemincludes a partial mirror which may be flat or have optical power. Inone embodiment, this partially transparent system includes a diffractiveoptical element. In one embodiment, this image is presented to the userthrough a direct view optical system. In one embodiment, this partiallytransparent system includes inclusions to reflect or scatter light.

In one embodiment of the field display 280, an additional displaysub-system is used to display images in the region of monovisionperipheral view 288. In one embodiment, this sub-system is an LED array.In one embodiment, this sub-system is an OLED array. In one embodiment,this display sub-system uses a scanned laser. In one embodiment, thissub-system uses an LCD panel. In one embodiment, this sub-system has nointermediate optical elements to manipulate the FOV or focus of theimage. In one embodiment, this sub-system has intermediate opticalelements. In one embodiment, these intermediate optical elements includea micro-lens array.

The image data displayed by the steerable foveal display 210 and fielddisplay 280 are generated by processing system 238. In one embodiment,the system includes an eye tracker 240. In one embodiment, an eyetracker 240 tracks the gaze vector of the user, e.g. where the eye islooking. In one embodiment, the eye tracking system is a camera-basedeye tracking system 240. Alternately, eye tracking system 240 may beinfrared laser based. Foveal position calculator 245 determines a centerof the user's field of view based on data from the eye tracking system240.

The processing system 238 in one embodiment further includes fovealposition validator 247 which validates the positioning of the positionelements 226, 236, to ensure that the displays 220, 230 are properlypositioned. In one embodiment, this includes re-evaluating the fovealdisplay location with respect to the center of the field of view of theuser's eye, in light of the movement of the foveal display. In oneembodiment, the foveal position validator 247 provides feedback toverify that the positioning element has reached its target location,using a sensing mechanism. The sensing mechanism may be a camera, in oneembodiment. The sensing mechanism may be gearing in one embodiment. Thesensing mechanism may be another type of sensor that can determine theposition of an optical element. In one embodiment, if the actualposition of the foveal display is not the target position, the fovealposition validator 247 may alter the display to provide the correctimage data. This is described in more detail below.

In one embodiment, eye movement classifier 260 can be used to predictwhere the user's gaze vector will move. This data may be used bypredictive positioner 265 to move the foveal display 220, 230 based onthe next position of the user's gaze vector. In one embodiment, smartpositioner 267 may utilize user data such as eye movement classificationand eye tracking to predictively position the displays 220, 230. In oneembodiment, smart positioner 267 may additionally use data aboutupcoming data in the frames to be displayed to identify an optimalpositioning for the displays 220, 230. In one embodiment, smartpositioner 267 may position the display 220, 230 at a position notindicated by the gaze vector. For example, if the displayed frame datahas only a small amount of relevant data (e.g. a butterfly illuminatedon an otherwise dark screen) or the intention of the frame is to causethe viewer to look in a particular position.

The processing system 238 may further include a cut-out logic 250.Cut-out logic 250 defines the location of the foveal display 220, 230and provides the display information with the cut-out to the associatedfield display 280. The field display 280 renders this data to generatethe lower resolution field display image including the cut out thecorresponding portion of the image in the field display. This ensuresthat there isn't interference between the foveal image and field image.In one embodiment, when there is a cut-out, blender logic 255 blends theedges of the cutout with the foveal image to ensure that the transitionis smooth. In another embodiment, the foveal display may be used todisplay a sprite, a brighter element overlaid over the lower resolutionfield image. In such a case, neither the cut out logic 250 nor blenderlogic 255 is necessary. In one embodiment, the cut out logic 250 andblender logic 255 may be selectively activated as needed.

In one embodiment, the system may synchronize the foveal display 210with an independent field display 280. In this case, in one embodiment,synchronization logic 272 synchronizes the displays. In one embodiment,the independent field display 280 is synchronized with the adjustablemirror, or other positioning element of the foveal display 210. Thisresults in the synchronization of the displays. The field display 280may receive positioning data. In one embodiment, there may not be acutout in this case.

In one embodiment, the processing system 238 may include an opticaldistortion system 275 for the foveal display 210 with distortion thatincreases from the center to the edge of the image. This intentionaldistortion would cause the pixels to increase in perceived size movingfrom the center of the foveal image to the edge. This change inperceived resolution would reduce the amount of processing required, asfewer pixels would be needed to cover the same angular area of thefoveal display image.

FIG. 5B shows an example of a distorted image with lower resolution asthe angle from the optical axis increases. The optical distortion mayhelp with the blending between the foveal display 210 and the fielddisplay 280. In another embodiment, the foveal display 210 including theoptical distortion system 275 could be used without a field display. Italso provides for an easier optical design, and saves processing on theblending.

In one embodiment, the variable resolution highly distorted image has alarge ratio between center and edge. The total FOV of this display wouldbe large (up to 180 degrees).

In one embodiment, roll-off logic 277 provides a roll-off at the edgesof the display. Roll-off in one embodiment may include resolutionroll-off (decreasing resolution toward the edges of the display area).In one embodiment, this may be implemented with magnification by theoptical distortion system 275. Roll-off includes in one embodimentbrightness and/or contrast roll off (decreasing brightness and/orcontrast toward the edges.) Such roll-off is designed to reduce theabruptness of the edge of the display. In one embodiment, the roll-offmay be designed to roll off into “nothing,” that is gradually decreasedfrom the full brightness/contrast to gray or black or environmentalcolors. In one embodiment, roll-off logic 277 may be used by the fovealdisplay 210 when there is no associated field display. In oneembodiment, the roll-off logic 297 may be part of the field display 280,when there is a field display in the system.

FIG. 3 illustrates one embodiment of the movement of the foveal imageover time, as the user's eye moves. In any time instance, there is asmall zone, to which the foveal image is displayed. The location of the5 degree display of high resolution (in this example) is focused on thecenter of the user's field of view. The low resolution field imageprovides a large field of view. But because the relative resolution ofthe eye outside the foveal area is lower, the user perceives thiscombination image, including the small high resolution foveal image andthe larger low resolution field image as high resolution across thelarge field of view.

FIG. 4A is a flowchart of one embodiment of utilizing the fovealdisplay. The process starts at block 410. In one embodiment, prior tothe start of this process the display system is fitted to the user. Thisinitial set-up includes determining the interpupillary distance (IPD)and any prescription needed, to ensure that the “baseline” display forthe user is accurate.

At block 415, the user's eyes are tracked. In one embodiment, an IRcamera is used for tracking eyes. In one embodiment, eye trackingidentifies the gaze vector of the user, e.g. where the user is focused.The eye tracking may identify left and right eye gaze vector/angle, andgaze center (derived from the L/R eye gaze vectors). The eye trackingmay determine the location (X, Y, Z) and orientation (roll, pitch, yaw)of the left and right eyes relative to a baseline reference frame. Thebaseline reference frame is, in one embodiment, established when thedisplay is initially fitted to the user and the user's interpupillarydistance, diopters, and other relevant data are established.

At block 420, the location of the fovea is determined based on the gazevector data. In one embodiment, the fovea location includes coordinates(X, Y, Z) and orientation (roll, pitch, yaw) for each eye.

At block 425, the process determines whether the foveal display shouldbe repositioned. This is based on comparing the current position of thefoveal display with the user's gaze vector or the intended position ofthe foveal image. If they are misaligned, the system determines that thefoveal display should be repositioned. If so, at block 430, the displayis repositioned. In one embodiment, if the foveal display is moved morethan a particular distance, the display is turned off during the move.This ensures that the user does not perceive the movement. In oneembodiment, the particular distance is more than 0.5 degrees. In oneembodiment, the foveal display is not turned off if the movement isoccurring while the user is blinking. Note that although the term“repositioning” is used, this does not generally mean that there is aphysical movement of the eye pieces. In one embodiment, a mirror orother optical elements which position the display are used to alter thecenter positioning of the foveal image. The process then continues toblock 435, whether or not the display was repositioned.

At block 435, optionally the system cuts out the portion of the fielddisplay that would be positioned in the same location as the fovealdisplay. This prevents the field display from interfering with thefoveal display. The cut-out, in one embodiment, is performed at therendering engine. In another embodiment, the foveal image may be asprite or other bright image element which does not need a cut-out to beclear. In that instance, this block may be skipped. In one embodiment,the cut-out is skipped if the user eye tracking indicates that theuser's gaze has moved substantially from the baseline reference. Thebaseline reference is the user's default gaze position, from which themovement of the gaze is tracked. A substantial movement from thebaseline reference means that the system cannot determine the user'scorrect gaze position. In this instance, in one embodiment, the fovealimage may be dropped, or the foveal display may be turned offmomentarily.

At block 440, in one embodiment, the edges between the foveal image andthe field image are blended. This ensures a smooth and imperceptibletransition between the field image and the foveal image. At block 445,the hybrid image is displayed to the user, incorporating the fovealdisplay and the field display. The process then returns to block 410 tocontinue tracking and displaying. Note that while the description talksabout a foveal image and a field image, the images contemplated includethe sequential images of video.

FIG. 4B illustrates one embodiment of the corrective actions which maybe taken when the display position validation indicates that the actuallocation of the foveal display does not match the intended location. Theprocess starts at block 450.

At block 452, the foveal display positioning is initiated. In oneembodiment, this corresponds to block 430 of FIG. 4A. Returning to FIG.4B, at block 454, the actual position of the foveal display is verified.In one embodiment, one or more sensors are used to determine thelocation and orientation of the foveal display. In one embodiment, thesensors may include cameras, mechanical elements detecting the positionof the adjustable mirror or other positioning element, etc.

At block 456 the process determines whether the foveal display iscorrectly positioned. Correct positioning has the foveal display in thecalculated location, to display the foveal image in the appropriatelocation for the user. If the foveal display is correctly positioned, atblock 464 the image is displayed. In one embodiment, this includesdisplaying a hybrid image including the foveal image in the calculatedlocation and the associated field display image. The process then endsat block 475.

If, at block 456, the process determines that the foveal display was notcorrectly positioned, the process continues to block 458.

At block 458, the process determines whether there is enough time forthe foveal display to be repositioned. This determination is based on adistance that needs to be moved, the speed of movement, and time untilthe next image will be sent by the processing system. In one embodiment,it also depends on the eye movement of the user. In one embodiment, thesystem preferentially moves the foveal display while the user isblinking, when no image is perceived. In one embodiment, therepositioning occurs within a blanking period of the display. Forexample, a movement of just one degree along one coordinate takes lesstime than moving the foveal display significantly and in threedimensions. If there is enough time, the process returns to block 452 toreposition the foveal display. Otherwise, the process continues to block460.

At block 460, the process determines whether the actual position of thefoveal display is within range of the intended position. In oneembodiment, “within range” in this context means that the system iscapable of adjusting the display for the difference. If it is withinrange, the process continues to block 462.

At block 462, the foveal image is adjusted for rendering in the actualposition, and the image is displayed at block 464. For example, in oneembodiment, the original calculated foveal image may be rendered in thewrong location if the position difference is very small, without causingvisual artifacts. In another embodiment, the foveal image may beadjusted to render appropriately at the actual location. For example,the foveal image may be cropped, brightened, distorted, contrastadjusted, chromatic coordinate (white point) adjusted, cropped, andlaterally shifted to account for the location difference. In oneembodiment, the radial location of the edge blending may be shifted orchanged. In one embodiment, the system may over-render, e.g. render 5.5degrees of visual image for a 5-degree foveal display, enabling a shiftof 0.5 degrees without needing re-rendering.

If the foveal display is not within range, at block 466, in oneembodiment the frame data is sent to the field display for rendering. Atblock 468, in one embodiment the foveal image is not displayed. In oneembodiment, the frame is dropped. In another embodiment, the fovealdisplay is turned off momentarily. In one embodiment, the foveal displayis not considered within range if the user eye tracking indicates thatthe user's gaze has moved too far outside of the baseline reference.

At block 470, the field display image is rendered, without the imagecut-out and without the display or rendering of the foveal image. Atblock 472, the field display image is displayed. The process then ends.

FIG. 5A illustrates one embodiment of the display including a fovealdisplay sub-system 510 and a field display sub-system 550. The fovealdisplay sub-system 510 includes a display panel 515 or another imagesource, and intermediate optics 520, in one embodiment. The output ofthe intermediate optics 520 is directed to an adjustable mirror 525 orother element which provides positioning. The adjustable mirror 525directs the image to partial mirror 530 and curved partial mirror 535,which direct the image toward the user's eye 590. In one embodiment, theadjustable mirror 525 may be replaced by a tunable prism, in which onesurface of a prism is moved to adjust the angle such as the Tunableprism TP-12-16 from OPTOTUNE™. In one embodiment, the adjustable mirror525 may be replaced by an acousto-optical modulator and mirror. In oneembodiment, each of these elements may be replaced with similarelements, which enable the selective movement of the high resolutiondisplay to be directed to align with the center of the field of view ofthe user's eye 590. The field display sub-system 550 in one embodimentincludes a projection sub-system 555 and a light guide 560. Alternativeembodiments may utilize different projection methods for the fielddisplay sub-system 550.

FIG. 5B illustrates one embodiment of roll-off which may be used toblend the foveal image with the field image. In one embodiment, thesystem resolution roll-off comprises magnifying the edges of the displayto show lower resolution data outside the foveal area. This alsoincreases the field of view. Magnification may be provided in variousways using hardware, software, or a combination. FIG. 16B illustrates anexemplary display 580 showing the distribution of the pixel density, asthe resolution rolls off. As can be seen in the center, the pixels areuniform size (illustrated by the central polygon 585). Toward the edgeof the display area the pixel size gets larger, and distorts. This canbe seen in left polygon 595. Because the distance between pixel edgesincreases both horizontally and vertically, in one embodiment the pixelswhich are horizontally and vertically removed from the central area aremore distorted, and larger, as can be seen in bottom polygon 1680. Notethat FIG. 5B illustrates a relatively small display and the ratiobetween the central polygon 585, and a corner polygon 595 may range fromgreater than 1 to less than or equal to 10.

FIG. 6 illustrates another embodiment of the display including a fovealsub-system 610 and a field display sub-system 650. In addition to thosetwo sub-systems, the embodiment of FIG. 6 includes a peripheral visiondisplay 670. The peripheral vision display in one embodiment is an OLEDdisplay.

FIG. 7 illustrates another embodiment of the display including a fovealdisplay sub-system 710 and a field display sub-system 750. The fielddisplay sub-system in this embodiment is an OLED with microlens array760.

FIG. 8 illustrates another embodiment of the display including a fovealdisplay sub-system 810 and an optional field display sub-system 850. Inthis embodiment, the foveal display sub-system 810 may be implemented inglasses or goggles, being worn by the user. The optional field displaysub-system 850 in one embodiment may be a display screen such as a TVmonitor 860. The field display sub-system 850 may be a modular elementwhich may be optionally attached to the glasses or goggles. In oneembodiment, the system may provide a high resolution image only throughthe foveal display sub-system 810. When the user does have the optionalfield display sub-system 850 available, the rendering system (not shown)can communicate with the foveal display sub-system 810 and field displaysub-system 850 provide a wider field of view. In one embodiment, in thisconfiguration, the foveal display sub-system may provide up to 20 degreefield of view.

FIG. 9 illustrates another embodiment of the display including a fovealdisplay sub-system 910 and a field display sub-system 950. In thisembodiment, the foveal display sub-system 910 comprises a light guide930 that has a FoV of 40-55°, coupled with a projector 920 which acts asa display panel, like an OLED microdisplay.

In one embodiment, the display panel 920 only sends a small image,associated with the area that covers the foveal region of the user'sfield of view instead of sending the full 40-55° image. The rest of thewaveguide 930, outside of the spot, would be transparent. Outside of thefoveal region, this could be filled in with a lower resolution fielddisplay 950, such as an OLED display 960.

FIG. 10 illustrates another embodiment of the display including a fovealdisplay sub-system 1010 and a field display sub-system 1050. In thisembodiment, the foveal display sub-system 1010 includes a display panel1015, intermediate optics 1020, an adjustable mirror 1025 directing thelight to an off-axis holographic optical element (HOE) 1030. The HOE1030 guides the light from the display 1015 to the user's eye. Theadjustable mirror 1025 provides the movement to enable the fovealdisplay sub-system 1010 to be correctly positioned. In one embodiment,the field display sub-system 1050 comprises a projection subsystem 1055and a light guide 1060.

FIG. 11 illustrates another embodiment of the display including a fovealdisplay sub-system 1110 and a field display sub-system 1150. In thisembodiment, the foveal display sub-system 1110 includes a display panel1115, intermediate optics 1120, an adjustable mirror 1125 directing thelight to a prism with an embedded partial mirror 1130. The light fromthe embedded partial mirror in the prism 1130 is reflected by a curvedpartial mirror 1140 to the user's eye. The adjustable mirror 1125provides the movement to enable the foveal display subsystem 1110 to becorrectly positioned. In one embodiment, the field display sub-system1150 comprises a projection subsystem 1155 and a light guide 1160.

FIG. 12 illustrates another embodiment of the display which provides aspatially multiplexed high resolution display and a low resolutiondisplay. In the embodiment of FIG. 12, the light is provided by a singledisplay panel 1210. The single display panel 1210 displays two separateimages, the foveal display portion and the field display portion. Thefoveal display portion passes through foveal display intermediate optics1220, an adjustable mirror 1230, and a partial mirror 1240 and curvedpartial mirror 1245. In one embodiment, the mirrors 1240, 1245 may bereplaced by another mechanism to redirect the light.

The field display image portion from the single display panel 1210 goesto field display intermediate options 1250, which passes them to lightguide 1260, in one embodiment. This enables a single display panel 1210to provide the data for both the foveal display and the field display,utilizing spatial multiplexing. In one embodiment, the relative size ofthe image on the display panel 1210 for the foveal display portion andthe field display portion are not identical. In one embodiment, thedisplay size is identical, but the field display intermediate optics1250 enlarge the portion of the image which will be utilized as thefield display.

FIGS. 13A and 13B illustrate one embodiment of a time multiplexeddisplay including a foveal image and a lower resolution field displayimage. The system utilizes a single display panel 1310 and a color orpolarization selective mirror 1325 which selectively sends the datathrough (for foveal image data) or reflects it to the field displayintermediate optics 1355. The display panel 1310 displays foveal imagedata and lower resolution field display data in a time multiplexed way,e.g. alternating frames at a speed fast enough to create two sets ofimages in human perception.

FIG. 13A illustrates the light path for the foveal image frame. The datagoes through foveal display intermediate optics 1320, and then isdirected through the color or polarization sensitive mirror 1325. It isreflected by adjustable mirror 1330. In one embodiment, a partial mirror1340 and curved partial mirror 1345 are used to direct the image to theuser's eye. In one embodiment, additional foveal display intermediateoptics 1320 may be positioned after the color or polarization selectivemirror 1325. Alternate configurations for directing the image may beused.

FIG. 13B illustrates the light path for the field display image data.The image data from the single panel display 1310 travels through fovealdisplay intermediate optics 1320 before being reflected by the color orpolarization selective mirror 1325, toward the field displayintermediate optics. In one embodiment, one or more redirecting mirrors1350 may be used to direct the light. From the field displayintermediate optics 1355 the light goes through a light guide 1360. Theoutput then passes through the curved partial mirror 1345 and partialmirror 1340 to the user's eye.

By switching the display rapidly between the foveal image and the fielddisplay image, the system displays the two images in a time multiplexedway so that both are simultaneously perceived by the user.

FIGS. 14A and 14B illustrate one embodiment of a foveal displaysub-system using a waveguide. This configuration of the foveal displaysub-system may be used in any of the embodiments described above, in oneembodiment. In one embodiment, the foveal image utilizes the displaypanel 1410. The output of display panel 1410 passes through optics 1420.Though optics 1420 is illustrated as a single lens, one of skill in theart would understand that any intermediate optics element may beincluded as optics 1420. The output of optics 1420 passes to steeringelement 1430, which steers it into the light guide in-couplers 1440.Steering element 1430 direct the light to the appropriate portion of thelight-guide in-coupler 1440. The image data then passes through thelight guide 1450, and out through light-guide out-coupler 1460 to theuser's eye. The steering element 1430 correctly directs the light forthe foveal image, adjusted to the user's eye position.

FIGS. 15A and 15B illustrate one embodiment of field display image usinga multi-layer light guide. This stacked waveguide may be used in theconfigurations described above for the field display. In this example,there are two waveguides, one for each portion of the field of view. Inanother embodiment, there may be four stacked waveguides.

The output of display panel 1510 pass through optics 1520. Though optics1520 is illustrated as a single lens, one of skill in the art wouldunderstand that any intermediate optics element may be included asoptics 1520. The output of optics 1520 pass to the light guidein-couplers 1540, 1545. In one embodiment, optics 1520 split the datafrom display panel 1510 based on color or polarization, and direct it toone of the light guide in-couplers 1540, 1545. In this example, the toplight guide 1550 is used for the first field of view portion of theimage, and the bottom light guide 1555 is used for the second field ofview portion of the image. The output from the foveal light guides 1550,1555 are directed by the light guide out coupler 1560, 1565 to theuser's eye.

FIG. 16A illustrates another embodiment of the display including afoveal display sub-system 1610 and a field display sub-system 1640. Thisconfiguration is similar to the configuration described above withrespect to FIG. 5, however instead of using an adjustable mirror, amovable display panel 1615 is used to position the foveal display forthe user's eye. This configuration for the movable element of the fovealdisplay sub-system may be utilized in the systems described above,replacing the adjustable mirror.

FIG. 16B illustrates another embodiment of the display including afoveal display sub-system 1650 and a field display sub-system 1690. Thisconfiguration is similar to the configuration described above withrespect to FIG. 5, however instead of using an adjustable mirror, atunable prism 1665 is used to position the foveal display for the user'seye. In this embodiment, one surface of the tunable prism is moved toadjust the angle such to position the foveal image. The tunable prismmay be tunable prism TP-12-16 from OPTOTUNE™. This configuration for themovable element of the foveal display sub-system may be utilized in thesystems described above, replacing the adjustable mirror with thetunable prism 1665. In another embodiment, the adjustable mirror 525 maybe replaced by an acousto-optical modulator and mirror. Thisconfiguration for the movable element of the foveal display sub-systemmay be utilized in the systems described above, replacing the adjustablemirror.

Note that the configurations shown in FIGS. 5 through 16A are presentedwith optics, and particular layouts. However, the design does notrequire the particular layouts, and additional optical elements may beutilized in the system. Furthermore, elements may be mixed and matchedbetween the configurations.

FIG. 17 is a flowchart of one embodiment of using the foveal displaywith an external display. An external display is a display notcontrolled by the same system as the foveal display. For example, theexternal display may be a projected system, for example in virtualreality (VR) cave or another environment which provides a field display.In one embodiment, the user may wear an augmented reality (AR) orvirtual reality (VR) headset, which interacts with the environment toprovide the hybrid display, with the AR/VR headset providing fovealdisplay, in addition to the field display provided by other systems.

The process starts at block 1710. At block 1715, a handshake isperformed between the foveal display system and the external displaysystem. In one embodiment, the handshake establishes that both systemsare capable of working together to provide the combination display. Inone embodiment, the handshake comprises setting up a connection betweenthe foveal display system and the field display system.

At block 1720, synchronization data is set from the external displaysystem. Because the foveal system is designed to be fully synchronizedwith the external system, in one embodiment, this synchronization signalprovides the frame data.

At block 1725, the positioning for the foveal display is determined. Asnoted above, this determination may be based on the user's gaze vector,predicted gaze, or smart positioning based on data from the frame beingdisplayed.

At block 1730, the process determines whether the foveal display shouldbe repositioned, to be displaying at the selected location. If so, atblock 1735, the positioning is triggered.

At block 1750, the foveal display is overlaid, to enhance the externaldisplay. In one embodiment, because the external display is separate, itdoes not include a cut-out logic. In another embodiment, there may be acut-out logic which keeps the system from rendering a portion of thelow-resolution image from the location at which the foveal display imageis shown.

At block 1760, a blur is applied to blend the edges between the fovealdisplay and field display images. The hybrid image including the fovealimage and the field image is displayed to the user, at block 1770. Inthis way, the user can have an enhanced viewing quality when entering aVR cave or other display environment which has a large field of view butfield display. The process then loops back to block 1720 to continue theprocess, until the video or other display ends.

FIG. 18 is a flowchart of one embodiment of using a foveal displaywithout an associated field display. In this case, the system providesonly a foveal display, without the field display discussed above.However, in one embodiment, the foveal display may have blending ormagnification applied to increase the field of view.

At block 1820, the process determines the position for the fovealdisplay, based on user data, or other data. The user data may includegaze vector, predicted gaze vector, etc. The external data may includeinformation about the image data which will be displayed.

At block 1830, the process determines whether the foveal display shouldbe repositioned. The display may not need to be repositioned formultiple frames because the user's gaze is unvarying. If the positionshould be altered, at block 1840 the foveal display is adjusted. In oneembodiment, the adjustment may include a steerable eye box to correctfor eye position. In one embodiment, the adjustment may include shiftingthe position of the display with respect to the foveal region of theuser's field of view. In one embodiment, the foveal display is turnedoff during the move, if the move is greater than a certain distance. Inone embodiment, the distance is more than 0.5 degrees. In oneembodiment, if the user is blinking during the move, the foveal displaymay not be turned off.

At block 1850, the foveal display is provided at the appropriateposition for the user.

At block 1860, in one embodiment, roll-off is provided at the edges ofthe display. Roll-off includes in one embodiment resolution roll-off(decreasing resolution toward the edges of the display area). Roll-offincludes in one embodiment brightness and/or contrast roll off(decreasing brightness and/or contrast toward the edges.) Such roll-offis designed to reduce the abruptness of the end of the display. In oneembodiment, the roll-off may be designed to roll off into “nothing,”that is gradually decreased from the full brightness/contrast to gray orblack or environmental colors.

In one embodiment, resolution roll-off comprises enlarging the pixelsize at the edges of the foveal display to better blend with the lowerresolution field display image outside the foveal area. This alsoincreases the field of view. Magnification may be provided in variousways using hardware, software, or a combination. FIG. 5B illustrates anexemplary display showing the distribution of the pixel density, as theresolution rolls off.

At block 1870, the appropriate gaze angle based correction is applied tothe image. As the gaze vector changes from the straight ahead, there isincreased distortion across the field of view. Gaze angle basedcorrection utilizes the known gaze angle, used for positioning, tocorrect for any distortion in software. The process then returns toblock 1820. In this way, the steerable foveal display may be used toprovide a steerable foveal display following the user's gaze, or othercues. In one embodiment, the foveal display may provide a variable fieldof view.

FIG. 19 is a flowchart of one embodiment of blending edges of the fovealdisplay. The process starts at block 1910. As discussed above, when thefoveal display is positioned with a field display, the edges between thedisplays are blended. This creates a continuous impression for the user.This process in one embodiment corresponds to block 440 of FIG. 4, andblock 1760 of FIG. 17.

At block 1920, the process identifies the edges of the foveal image. Theedges, in one embodiment, are defined by the field of view available tothe foveal display. In another embodiment, the foveal display maydisplay a field of view less than the maximum it can display.

At block 1930, the process determines the best blending technique, andapplies it. In one embodiment, the blending techniques may includeblending using an alpha mask, dithered blend, interlacing pixels, colorbased alpha channel blending, pixel based alpha channel blending,multi-sample antialiasing (MSAA), temporal filtering blending, and/orother blending techniques.

At block 1950, the process determines whether other techniques should beapplied. If so, at block 1960 the next technique is selected, and theprocess returns to block 1940. If not, the process ends at block 1970.As noted above, in one embodiment this process is invoked with eachframe that includes a high resolution foveal display image and a lowerresolution field display image superimposed. In one embodiment, when thefoveal display shows a sprite or other image element that issuperimposed on a background, no blending may be applied.

FIG. 20 is a flowchart of one embodiment of using eye movementclassification. Eye movement classification is used to predict thefuture location of the user's eye for positioning the foveal display.The process starts at block 2010. At block 2015, the location of thefoveal region of the user's field of view is determined 2015. At block2020, the user's eye movement is classified. FIG. 21 illustrates someexemplary eye movements that may be identified. The eye movementsinclude fixated, blinking, micro-saccade, slow pursuit, and fastmovement/saccade. In one embodiment, in addition to the eye movement,the head position may be used in classifying the eye movement forpredictive purposes. These types of eye movements are known in the art.

At block 2025, the process determines an appropriate response to the eyemovement. The responses may include altering the position of thedisplay, altering the field of view, altering the resolution, alteringdepth data (which may depend on 3D gaze vector), altering theconvergence point. The determination may be based on predicting asubsequent location of the user's gaze vector based on the eye movementclassification.

At block 2030, the process determines whether the foveal display shouldbe changed. If so, at block 2035, the foveal display is altered. Asnoted above, the alteration may include changes in position, field ofview, resolution, etc.

At block 2040 the process determines whether the field display should bechanged based on the analysis. If so, at block 2045 the field display ischanged. In one embodiment, the field display may be changed by changingresolution, depth data, convergence point, etc. In one embodiment, thefield display is not steerable, but other changes may be made.

At block 2050, the edges are blurred between the foveal display and thefield display images. At block 2060 the hybrid image is displayed to theuser. The process then returns to block 2015 to continue processing thenext image. Note that this process, in one embodiment, occurs veryquickly so that the evaluation is made for each frame prior to itsdisplay.

FIG. 22 is a flowchart of one embodiment of smart positioning. Theprocess starts at block 2210. This process may be used when a system isdesigned to utilize positioning not just based on the gaze vector of theuser.

At block 2215, the user's eyes are tracked. In one embodiment, theuser's head movement may also be tracked. This is useful in predictingthe user's eye movements based on the vestibular ocular reflex. Headmovement and eye movement may be combined to determine the position andorientation of each eye.

At block 2220, external data is received. This external data may includea highlighted element that should be shown in high resolution, using thefoveal display, a location which the user's eyes should be guided, oranother external factor. In one embodiment, the foveal display may bepointed to a relevant element that is not at the user's gaze vector. Forexample, when there is a dark screen and only one element of interest,the high resolution foveal display is best deployed at the interestingelement. As another example, if the majority of the screen isdeliberately blurry, but there is some portion with writing or otherfine detail content, that may be the place to deploy the foveal display.Other reasons to position the display may be used.

At block 2225, the optimal positioning and configuration is determinedfor the foveal display based on external data and user data. As notedabove, the user data includes the user's eye and head positioning. Theexternal data is independent of the user, and reflects information aboutthe frame being displayed, in one embodiment. In one embodiment, unlessthere is external data retargeting the foveal display, the defaultconfiguration is to position it at foveal center for the user. However,based on external information, this may be changed for certain framesand content.

At block 2230, the process determines whether the foveal display shouldbe altered. The change may be a change in position, resolution, focaldistance, etc. If so, at block 2235, the display is changed.

At block 2240, the process determines whether the field display portionshould be altered. The change may be a change in resolution, brightness,contrast, etc. If so, at block 2245, the display is changed.

At block 2250, the edges between the foveal display and the fielddisplay images are blended, and at block 2255 the combination image isdisplayed. The process then returns to block 2215.

Although the above processes are illustrated in flowchart form, one ofskill in the art would understand that this is done for simplicity. Theorder of the various elements need not remain the same, unless there isa dependency between the elements. For example, the adjustment of thefoveal display and field displays may be done in any order. The trackingof the user's eyes and head may be done continuously. The system mayreceive external data when it is available, rather than continuously orat a particular time in the process. Other such adjustments to theflowcharts are within the scope of this invention.

FIG. 23 is a block diagram of one embodiment of a computer system thatmay be used with the present invention. It will be apparent to those ofordinary skill in the art, however that other alternative systems ofvarious system architectures may also be used.

The data processing system illustrated in FIG. 23 includes a bus orother internal communication means 2340 for communicating information,and a processing unit 2310 coupled to the bus 2340 for processinginformation. The processing unit 2310 may be a central processing unit(CPU), a digital signal processor (DSP), or another type of processingunit 2310.

The system further includes, in one embodiment, a random access memory(RAM) or other volatile storage device 2320 (referred to as memory),coupled to bus 2340 for storing information and instructions to beexecuted by processor 2310. Main memory 2320 may also be used forstoring temporary variables or other intermediate information duringexecution of instructions by processing unit 2310.

The system also comprises in one embodiment a read only memory (ROM)2350 and/or static storage device 2350 coupled to bus 2340 for storingstatic information and instructions for processor 2310. In oneembodiment, the system also includes a data storage device 2330 such asa magnetic disk or optical disk and its corresponding disk drive, orFlash memory or other storage which is capable of storing data when nopower is supplied to the system. Data storage device 2330 in oneembodiment is coupled to bus 2340 for storing information andinstructions.

The system may further be coupled to an output device 2370, such as acathode ray tube (CRT) or a liquid crystal display (LCD) coupled to bus2340 through bus 2360 for outputting information. The output device 2370may be a visual output device, an audio output device, and/or tactileoutput device (e.g. vibrations, etc.)

An input device 2375 may be coupled to the bus 2360. The input device2375 may be an alphanumeric input device, such as a keyboard includingalphanumeric and other keys, for enabling a user to communicateinformation and command selections to processing unit 2310. Anadditional user input device 2380 may further be included. One such userinput device 2380 is cursor control device 2380, such as a mouse, atrackball, stylus, cursor direction keys, or touch screen, may becoupled to bus 2340 through bus 2360 for communicating directioninformation and command selections to processing unit 2310, and forcontrolling movement on display device 2370.

Another device, which may optionally be coupled to computer system 2300,is a network device 2385 for accessing other nodes of a distributedsystem via a network. The communication device 2385 may include any of anumber of commercially available networking peripheral devices such asthose used for coupling to an Ethernet, token ring, Internet, or widearea network, personal area network, wireless network or other method ofaccessing other devices. The communication device 2385 may further be anull-modem connection, or any other mechanism that provides connectivitybetween the computer system 2300 and the outside world.

Note that any or all of the components of this system illustrated inFIG. 23 and associated hardware may be used in various embodiments ofthe present invention.

It will be appreciated by those of ordinary skill in the art that theparticular machine that embodies the present invention may be configuredin various ways according to the particular implementation. The controllogic or software implementing the present invention can be stored inmain memory 2320, mass storage device 2330, or other storage mediumlocally or remotely accessible to processor 2310.

It will be apparent to those of ordinary skill in the art that thesystem, method, and process described herein can be implemented assoftware stored in main memory 2320 or read only memory 2350 andexecuted by processor 2310. This control logic or software may also beresident on an article of manufacture comprising a computer readablemedium having computer readable program code embodied therein and beingreadable by the mass storage device 2330 and for causing the processor2310 to operate in accordance with the methods and teachings herein.

The present invention may also be embodied in a handheld or portabledevice containing a subset of the computer hardware components describedabove. For example, the handheld device may be configured to containonly the bus 2340, the processor 2310, and memory 2350 and/or 2320.

The handheld device may be configured to include a set of buttons orinput signaling components with which a user may select from a set ofavailable options. These could be considered input device #1 2375 orinput device #2 2380. The handheld device may also be configured toinclude an output device 2370 such as a liquid crystal display (LCD) ordisplay element matrix for displaying information to a user of thehandheld device. Conventional methods may be used to implement such ahandheld device. The implementation of the present invention for such adevice would be apparent to one of ordinary skill in the art given thedisclosure of the present invention as provided herein.

The present invention may also be embodied in a special purposeappliance including a subset of the computer hardware componentsdescribed above, such as a kiosk or a vehicle. For example, theappliance may include a processing unit 2310, a data storage device2330, a bus 2340, and memory 2320, and no input/output mechanisms, oronly rudimentary communications mechanisms, such as a small touch-screenthat permits the user to communicate in a basic manner with the device.In general, the more special-purpose the device is, the fewer of theelements need be present for the device to function. In some devices,communications with the user may be through a touch-based screen, orsimilar mechanism. In one embodiment, the device may not provide anydirect input/output signals, but may be configured and accessed througha website or other network-based connection through network device 2385.

It will be appreciated by those of ordinary skill in the art that anyconfiguration of the particular machine implemented as the computersystem may be used according to the particular implementation. Thecontrol logic or software implementing the present invention can bestored on any machine-readable medium locally or remotely accessible toprocessor 2310. A machine-readable medium includes any mechanism forstoring information in a form readable by a machine (e.g. a computer).For example, a machine readable medium includes read-only memory (ROM),random access memory (RAM), magnetic disk storage media, optical storagemedia, flash memory devices, or other storage media which may be usedfor temporary or permanent data storage. In one embodiment, the controllogic may be implemented as transmittable data, such as electrical,optical, acoustical or other forms of propagated signals (e.g. carrierwaves, infrared signals, digital signals, etc.).

In the foregoing specification, the invention has been described withreference to specific exemplary embodiments thereof. It will, however,be evident that various modifications and changes may be made theretowithout departing from the broader spirit and scope of the invention asset forth in the appended claims. The specification and drawings are,accordingly, to be regarded in an illustrative rather than a restrictivesense.

1. A hybrid display system comprising: a field display having amonocular field of view of at least 40 degrees; a foveal display havinga monocular field of view of at least 1 degree, positioned within ascannable field of view spanning at least 20 degrees, the foveal displaypositioned for a user within the field of view of the field display,thereby presenting a hybrid display including image data from each ofthe field display and the foveal display.
 2. The hybrid display systemof claim 1, further comprising: wherein the foveal display isimplemented in a wearable display foveal display.
 3. The hybrid displaysystem of claim 1, further comprising a foveal position validator, toverify an actual position of the foveal display and to adjust fovealdisplay image data when the actual position is not an intended position.4. The hybrid display system of claim 1, further comprising: the fovealdisplay positioned based on a prediction of a future eye position. 5.The hybrid display system of claim 3, further comprising: an eye trackerto track a user's eye position and orientation; and an eye movementclassifier to identify eye movement types, the eye movementclassification used to provide the prediction for predictivelypositioning the foveal display.
 6. The hybrid system of claim 1, furthercomprising: a cut-out logic to cut out a portion of a field displayimage from the field display at a position corresponding to a fovealdisplay image.
 7. The hybrid display system of claim 6, furthercomprising: a blending logic to blend edges where the foveal displayimage and the field display image meet.
 8. The hybrid display system ofclaim 7, wherein the blending comprises one or more of alpha mask,dithered blend, interlacing pixels, color based alpha channel blending,pixel based alpha channel blending, multi-sample antialiasing (MSAA),and temporal filtering blending.
 9. The hybrid display system of claim1, wherein the field display is an external display and furthercomprising: a synchronization logic to synchronize display between thefoveal display and the field display.
 10. The hybrid display system ofclaim 1, further comprising: position elements for the foveal display toenable movement of the steerable foveal display.
 11. The hybrid displaysystem of claim 10, wherein the positionable elements include one ormore of: adjustable mirror, tunable prism, acousto-optical modulator,adjustable display panel, a curved mirror, a diffractive element, and aFresnel reflector.
 12. A display system comprising: a foveal displayhaving a monocular field of view of at least 1 degree, positioned withina scannable field of view of at least 20 degrees, the foveal displaypositioned for a user.
 13. The display system of claim 12, furthercomprising: a field display having a monocular field of view of at least30 degrees displaying a field display image, the field display imageoverlapping the foveal display image; such that a combination of thefoveal display and the field display provides a large field of view anda perceived high resolution.
 14. The display system of claim 13, furthercomprising: a cut-out logic to cut out a portion of a field displayimage at a position corresponding to the position of a foveal displayimage.
 15. The display system of claim 14, further comprising a blendinglogic to blend edges where the foveal display image and the fielddisplay image meet.
 16. The display system of claim 15, wherein theblending comprises one or more of alpha mask, dithered blend,interlacing pixels, color based alpha channel blending, pixel basedalpha channel blending, multi-sample antialiasing (MSAA), and temporalfiltering blending.
 17. The display system of claim 12, furthercomprising: a synchronization logic to synchronize display between thefoveal display and an external display providing a field display. 18.The display system of claim 12, further comprising: an eye tracker totrack a user's eye position and orientation; and an eye movementclassifier to identify eye movement types, the eye movementclassification used to predictively position the foveal display.
 19. Thedisplay system of claim 12, further comprising positioning elements forthe foveal display to enable movement of the steerable foveal display.20. The display system of claim 19, wherein the positioning elementsinclude one or more of: adjustable mirror, tunable prism,acousto-optical modulator, adjustable display panel, a curved mirror, adiffractive element, and a Fresnel reflector.